System and method for adjusting recording modes for driver facing cameras

ABSTRACT

A system for recording vehicle occupants and their immediate environment includes a driver-facing camera for capturing video data, an input that receives data from one or more sensors configured to detect vehicle related data and a processor. The processor is configured to embed, based on the sensor data, at least one indicator in a portion of the video data. The indicator designates that the portion of the video data is to be obscured during playback of the video data.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of the pending U.S. patentapplication Ser. No. 16/664,626 entitled “System and Method forAdjusting Recording Modes for Driver Facing Camera” filed Oct. 25, 2019,the entire disclosure of which is incorporated fully herein byreference.

FIELD OF THE INVENTION

The invention relates to adjusting recording modes of driver facingcameras and, in particular, to adjusting recording modes of driverfacing cameras in response to the detection of certain events, whichallows for the recording of driver activity during periods and at levelsof detail that minimize the invasion of driver privacy.

BACKGROUND

Current methods of capturing driving data include capturing video datavia one or more driver facing cameras. However, such driver facingcameras have the potential to invade driver privacy, particularly duringrest periods in which the driver may be located in a bunk area visibleto the driver facing cameras. Recording during under these circumstancesis potentially intrusive and is often unnecessary, and may lead to aninsufficiently rested and resentful driver.

The current approach for limiting the potential intrusion of driverprivacy is to simply turn off the driver facing camera. However, asidefrom the inconvenience of having to manually operate the driver facingcamera, this known approach can be overly restrictive in that evenpermissible data is not collected while the driver facing camera isturned off. Moreover, even if non-video data continues to be collectedwhile the driver facing camera is turned off, there will still be gapsin or desynchronization of the data since there is no correspondingvideo data to associate with any other event data that may have beencollected.

As such, there is a need in the art for a system and method thatovercomes the aforementioned drawbacks.

SUMMARY OF THE INVENTION

In one embodiment of the invention, a system for recording vehicleoccupants and their immediate environment using multiple recording modesof a camera is provided. The system includes a driver-facing camerahaving a plurality of recording modes. At least one of the recordingmodes is preferably an obscuring mode in which the image provided by thevideo data is partially or wholly obscured. The system also includes aninput configured to receive vehicle related data from one or moresensors that detect the vehicle related data, including one or more of:time, driver and/or passenger location in the vehicle, vehicle behavior,parking brake status, ignition status. A processor is also provided,which is configured to determine a mode-changing event based on thedetected vehicle related data, and to alter the recording mode of thedriver-facing camera in response to determining the mode-changing event.

Other objects, advantages and novel features of the present inventionwill become apparent from the following detailed description of one ormore preferred embodiments when considered in conjunction with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram that illustrates a vehicle-based computersystem configured to implement one or more aspects of the invention;

FIG. 2 is a schematic diagram of the operation of the systems andmethods according to one or more aspects of the invention;

FIG. 3 illustrates at least one embodiment of a process for implementingone or more aspects of the invention;

FIG. 4 is a schematic diagram of the operation of the systems andmethods according to one or more aspects of the invention;

FIG. 5 is a schematic diagram of the operation of the systems andmethods according to one or more aspects of the invention.

FIG. 6A to FIG. 6G illustrate various recording modes according to oneor more aspects of the invention.

DESCRIPTION OF EXEMPLARY EMBODIMENTS OF THE INVENTION

In the following description of the present invention reference is madeto the accompanying figures which form a part thereof, and in which isshown, by way of illustration, exemplary embodiments illustrating theprinciples of the present invention and how it is practiced. Otherembodiments can be utilized to practice the present invention andstructural and functional changes can be made thereto without departingfrom the scope of the present invention

In certain embodiments, the invention may be implemented by anon-vehicle event detection and reporting system that may include one ormore driver facing cameras that are configured such that the field ofview of the camera(s) captures a view the driver of the vehicle, and/ora view of other areas of the cabin, such as the driver controls of thevehicle while driving. Still other embodiments may include camerasconfigured to capture other scenes relative to the vehicle. Forinstance, embodiments may include cameras configured to capture thescene in front of the vehicle, behind the vehicle, to either side of thevehicle, etc.

The event detection and reporting system may be further configured tocollect and provide non-video data, including non-video event-based datacorresponding to a detected driver or vehicle event that occurred at aparticular point in time during a driving excursion. Such event-baseddata can include data collected from components of, or componentsinteracting with, the event detection and reporting system.

These components can detect, in real time, driver or vehicle-relatedevents that happen over the course of a driving excursion, or evenoutside of the driving excursion. The components can report such eventsto the detection and reporting system. Examples of events that may bedetected and/or reported to/collected by the event detection andreporting system in real time include safety events, for example andwithout limitation, excessive acceleration, excessive braking, exceedingspeed limit, excessive curve speed, excessive lane departure, lanechange without turn signal, loss of video tracking, LDW system warning,following distance alert, forward collision warning, collisionmitigation braking, collision occurrence, etc., and non-safety events,for example and without limitation, the driver logging in/out of avehicle telematics system, the driver/passenger entering/leaving thevehicle, the driver/passenger occupying/vacating the bunk area, thedriver occupying/vacating the driver seat, the vehicle engine beingon/off, the vehicle gear being in park/drive, the parking brake beingon/off, etc. Non-safety events may also include theft events, forexample and without limitation, the presence of an unauthorized occupantaccessing the vehicle, etc.

In accordance with an embodiment, the event detection and reportingsystem may use data collected directly from vehicle components (e.g.,devices, sensors, or systems), and data collected from an analysis ofvehicle video, to generate event datasets that correspond in time withone or more detected events. Event data generated for a detected eventmay be associated with captured video frames whose timeline spans oroverlaps the time when the event was detected/collected. Event datagenerated from an event determined from processing of captured vehiclevideo may at least be associated with the video from which it wasgenerated, but may also be associated with other captured video frameswhose timelines span or overlap the time when the event wasdetected/collected (in these scenarios, the time may be calculated basedon the video frame or frames from which the event object was derived).

The particular invention disclosed and claimed herein relates to asystem and method for adjusting recording modes of driver facing camerasin response to the detection of mode-change events. In certainembodiments, this is achieved by entering a different mode of recordingin response to detecting mode-change events, based on, for example,detected driver activity, vehicle state, telematics information, and/orsafety events. This different mode of recording is characterized by thefact that the video data of the driver facing camera is altered in amanner which obscures the recorded image data, in whole or in part, oris otherwise not recorded. For example, modern imager chips havesettings that control the video signal produced, including gain,exposure time, black clamping level, etc. One aspect of the invention isto provide a mode of recording in which one or more of these setting ismodified (e.g., very low or very high gain, for example) in response tothe detection.

Referring first to FIG. 1, by way of overview a schematic block diagramis provided illustrating details of an event detection and reportingsystem configured to be used in accordance with one or more exemplaryembodiments of the invention. The in-vehicle event detection andreporting system 100 may be adapted to detect a variety of operationalparameters and conditions of the vehicle and the driver's interactiontherewith and, based thereon, to determine if a driving or vehicle eventhas occurred (e.g., if one or more operational parameter/conditionthresholds has been exceeded). Data related to detected events (i.e.,event data) may then be stored and/or transmitted to a remotelocation/server, as described in more detail below.

The event detection and reporting system 100 of FIG. 1 may include oneor more devices or systems 114 for providing vehicle-related input dataindicative of one or more operating parameters or one or more conditionsof a commercial vehicle. Alternatively, the event detection andreporting system 100 may include a signal interface for receivingsignals from the one or more devices or systems 114, which may beconfigured separate from system 100. For example, the devices 114 may beone or more sensors, such as but not limited to, one or more wheel speedsensors 116, one or more acceleration sensors such as multi-axisacceleration sensors 117, a steering angle sensor 118, a brake pressuresensor 119, one or more vehicle load sensors 120, a yaw rate sensor 121,a lane departure warning (LDW) sensor or system 122, one or more enginespeed or condition sensors 123, and a tire pressure (TPMS) monitoringsystem 124. The event detection and reporting system 100 may alsoutilize additional devices or sensors, including for example a forwarddistance sensor 160 and a rear distance sensor 162 (e.g., radar, lidar,etc.). Other sensors and/or actuators or power generation devices orcombinations thereof may be used of otherwise provided as well, and oneor more devices or sensors may be combined into a single unit as may benecessary and/or desired.

The event detection and reporting system 100 may also include brakelight(s) 166 and/or notification device 164, and may be usable toprovide headway time/safe following distance warnings, lane departurewarnings, and warnings relating to braking and or obstacle avoidanceevents.

The event detection and reporting system 100 may also include a logicapplying arrangement such as a controller or processor 130 and controllogic 131, in communication with the one or more devices or systems 114.The processor 130 may include one or more inputs for receiving inputdata from the devices or systems 114. The processor 130 may be adaptedto process the input data and compare the raw or processed input data toone or more stored threshold values or desired averages, or to processthe input data and compare the raw or processed input data to one ormore circumstance-dependent desired value.

The processor 130 may also include one or more outputs for delivering acontrol signal to one or more vehicle systems 133 based on thecomparison. The control signal may instruct the systems 133 to provideone or more types of driver assistance warnings (e.g., warnings relatingto braking and or obstacle avoidance events) and/or to intervene in theoperation of the vehicle to initiate corrective action. For example, theprocessor 130 may generate and send the control signal to an engineelectronic control unit or an actuating device to reduce the enginethrottle 134 and slow the vehicle down. Further, the processor 130 maysend the control signal to one or more vehicle brake systems 135, 136 toselectively engage the brakes (e.g., a differential braking operation).A variety of corrective actions may be possible and multiple correctiveactions may be initiated at the same time.

The event detection and reporting system 100 may also include a memoryportion 140 for storing and accessing system information, such as forexample the system control logic 131. The memory portion 140, however,may be separate from the processor 130. The sensors 114 and processor130 may be part of a preexisting system or use components of apreexisting system.

The event detection and reporting system 100 may also include a sourceof vehicle-related input data 142 indicative of aconfiguration/condition of a commercial vehicle. The processor 130 maysense or estimate the configuration/condition of the vehicle based onthe input data, and may select a control tuning mode or sensitivitybased on the vehicle configuration/condition. The processor 130 maycompare the operational data received from the sensors or systems 114 tothe information provided by the tuning.

In addition, the event detection and reporting system 100 is operativelycoupled with one or more driver facing imaging devices, shown in theexample embodiment for simplicity and ease of illustration as a singledriver facing camera 145 that is trained on the driver and/or trained onthe interior of the cab of the commercial vehicle. However, it should beappreciated that one or more physical video cameras may be disposed onthe vehicle such as, for example, a video camera on each corner of thevehicle, one or more cameras mounted remotely and in operativecommunication with the event detection and reporting system 100 such asa forward facing camera 146 to record images of the roadway ahead of thevehicle. In the example embodiments, driver data can be collecteddirectly using the driver facing camera 145 in accordance with adetected driver head position, hand position, or the like, within thevehicle being operated by the vehicle. In addition, driver identity canbe determined based on facial recognition technology and/or body/posturetemplate matching.

Still yet further, the event detection and reporting system 100 may alsoinclude a transmitter/receiver (transceiver) module 150 such as, forexample, a radio frequency (RF) transmitter including one or moreantennas 152 for wireless communication of the automated controlrequests, GPS data, one or more various vehicle configuration and/orcondition data, or the like between the vehicles and one or moredestinations such as, for example, to one or more services (not shown)having a corresponding receiver and antenna. The transmitter/receiver(transceiver) module 150 may include various functional parts of subportions operatively coupled with a platoon control unit including forexample a communication receiver portion, a global position sensor (GPS)receiver portion, and a communication transmitter. For communication ofspecific information and/or data, the communication receiver andtransmitter portions may include one or more functional and/oroperational communication interface portions as well.

The processor 130 is operative to combine selected ones of the collectedsignals from the sensor systems described herein into processed datarepresentative of higher level vehicle condition data such as, forexample, data from the multi-axis acceleration sensors 117 may becombined with the data from the steering angle sensor 118 to determineexcessive curve speed event data. Other hybrid event data relatable tothe vehicle and driver of the vehicle and obtainable from combining oneor more selected raw data items from the sensors includes, for exampleand without limitation, excessive braking event data, excessive curvespeed event data, lane departure warning event data, excessive lanedeparture event data, lane change without turn signal event data, lossof video tracking event data, LDW system disabled event data, distancealert event data, forward collision warning event data, haptic warningevent data, collision mitigation braking event data, ATC event data, ESCevent data, RSC event data, ABS event data, TPMS event data, enginesystem event data, average following distance event data, average fuelconsumption event data, average ACC usage event data, and late speedadaptation (such as that given by signage or exiting).

The event detection and reporting system 100 of FIG. 1 is suitable forexecuting embodiments of one or more software systems or modules thatperform vehicle brake strategies and vehicle braking control methodsaccording to the subject application. The example event detection andreporting system 100 may include a bus or other communication mechanismfor communicating information, and a processor 130 coupled with the busfor processing information. The computer system includes a main memory140, such as random access memory (RAM) or other dynamic storage devicefor storing instructions and loaded portions of the trained neuralnetwork to be executed by the processor 130, and read only memory (ROM)or other static storage device for storing other static information andinstructions for the processor 130. Other storage devices may alsosuitably be provided for storing information and instructions asnecessary or desired.

Instructions may be read into the main memory 140 from anothercomputer-readable medium, such as another storage device of via thetransceiver 150. Execution of the sequences of instructions contained inmain memory 140 causes the processor 130 to perform the process stepsdescribed herein. In an alternative implementation, hard-wired circuitrymay be used in place of or in combination with software instructions toimplement the invention. Thus implementations of the example embodimentsare not limited to any specific combination of hardware circuitry andsoftware.

Referring now to FIGS. 2 and 4-5, simplified schematic block diagramsare provided illustrating details of the adjustment of the recordingmodes of driver facing camera system 200 in response to the detection ofmode-change events via the event detection and reporting system of FIG.1.

The driver facing camera system 200 includes one or more sensors,devices or systems 114 for providing input data, e.g., vehicle relateddata, indicative of one or more operating parameters or one or moreconditions of a commercial vehicle, as discussed with reference to FIG.1.

The driver facing camera system 200 also includes the driver facingcamera 145 via which driver data, which is another form ofvehicle-related data, including but not limited to, driver headposition, hand position, location within the vehicle, identity, or thelike, can be detected and collected. Driver data can be determined basedon facial recognition technology and/or body/posture template matching.Passenger or other occupant data can be similarly detected and collectedusing the driver facing camera 145.

The driver facing camera can also function as a motion sensor, whichdetects activity through persistent/sufficient gain/exposure changes inthe camera according to techniques known in the art. As such, the driverfacing camera can act as another sensor 114 for providing vehiclerelated data. Situation dependent image control weighting may beutilized under such circumstances to identify areas of the camera imagewhere relevant activity is to be detected. In particular, the image maybe divided into a matrix of rectangular image areas, whose effect on thegain/exposure levels may be adjustably weighted to identify image areasfor activity detection, i.e., motion detection. For example, theimager's gain/exposure dependence on the image areas corresponding tothe bunk region in the image can be set to zero—so that activity in thebunk region does not register as an exposure change and therefore doesnot register as motion. In other words, the driver facing cameraexposure control may be “blind” to changes in such image areas.

The driver facing camera system also includes a video-clip memory 140-a(FIG. 4) and a DVR memory 140-b (FIG. 2), each of which may comprise oneor more distinct physical memories or memory locations within one ormore common physical memories. In at least one embodiment, thevideo-clip memory 140-a and the DVR memory 140-b form part of the mainmemory 140 of the event detection and reporting system 100.

The driver facing camera 145, the processor 130 and the video-clipmemory 140-a form a video-clip recorder 204 (FIG. 4), which is generallyconfigured to record to the video-clip memory 140-a video data that hasbeen captured by the driver facing camera 145 during a window of time inwhich a recording triggering event, e.g., a detected safety event ortheft event, occurred (e.g., 10 seconds before to 10 seconds after thedetected event).

In at least one embodiment, the video-clip recorder is enabled by theprocessor 130 when one or more of the following conditions is detected:(a) the driver is in the driver seat, (b) the engine is on, (c) a safetyevent has occurred, and (d) an unauthorized occupant is present withoutthe driver. Otherwise, while the driver facing camera may be enabled forother purposes, e.g., motion detection, video-clip recording ispreferably not enabled.

The driver facing camera 145, the processor 130 and the DVR memory 140-bform a DVR recorder 202 (FIG. 2), which is generally configured torecord to the DVR memory 140-b video data on a continuous basis,irrespective of whether a safety event or theft event has been detectedor not.

At any given time, the video-clip recorder 204 and/or the DVR recorder202 can record to the respective memory 140-a, 140-b in accordance withone or more of a plurality of available recording modes, including:normal and unobscured recording modes. In the normal recording mode(s),the recorder records video data to the memory without obscuring theimage, whereas in the obscured recording modes, the video data isrecorded to the memory such that, on visual playback of the video data,the displayed image is visually obscured. FIG. 6A illustrates the normalor unobscured recording mode. FIGS. 6B-6G illustrate various obscuredrecording modes.

In some embodiments, different recording modes may be enabled fordifferent image areas. In particular, the image may be divided into amatrix of rectangular image areas 602, and imaging parameters of eachimage area 602 may be controlled to obscure and un-obscure respectiveimage areas in accordance with the desired recording mode. In someembodiments, the imaging parameters are adjustably weighted so as toobscure and un-obscure respective image areas 602. Control of theimaging parameters may be done via the processor 130 and/or the driverfacing camera.

Accordingly, video data may be selectively obscured and/or unobscuredfor the image areas, i.e., areas-of-interest, that include one or moreof the following vehicle regions: the driver seat, the passenger seat,between passenger and driver seats, the bunk area, the doors, thevehicle controls, the steering wheel, and the like. Video data may alsobe selectively obscured and/or unobscured for areas areas-of-interestthat include the driver and/or passenger, or portions thereof, includingbut not limited to faces and other body parts. Transitions, spatialand/or temporal, between unobscured and obscured image areas, as well asbetween differently obscured image areas, may be gradual or sudden.Moreover, the obscuring may be reversible or irreversible.

In some embodiments, obscuring may be based, at least in part, on apersonal preference of the driver, which may be tied to the driver'stelematic log-in information, the facial recognition of the driver, orother type of recognition of the driver by the system. In suchembodiments, a default mode may be enabled where the driver is notrecognized, and/or there is no associated personal preference.

Exemplary approaches to obscuring/un-obscuring the video data inaccordance with various recording modes will now be discussed.

FIG. 6B illustrates a blacked-out mode, in which the brightness level ofthe image area can be set such that on playback the image area isvisually unintelligible or of reduced visual intelligibility. Forexample, the brightness levels can be set to zero, resulting in ablacked out image area.

FIG. 6C illustrates a low-resolution mode, in which the spatialresolution levels of the image area can be set such that on playback theimage area is visually unintelligible or of reduced visualintelligibility. For example, the resolution levels can be set to resultin an image of such low-resolution as to be unintelligible.

FIG. 6D illustrates a darkened mode, in which the grey levels of theimage area can be set or remapped via mathematical transformation suchthat on playback the image area is visually unintelligible or of reducedvisual intelligibility. For example, the grey levels can be set to zero,resulting in a shadowy blind spot. Alternatively, the grey levels can beset, limited or remapped to a set of low non-zero values, resulting in a“lost in deep shadow” image.

FIG. 6E illustrates an encrypted mode, in which the video data of theimage areas can be encrypted such that on playback the image area isvisually unintelligible or of reduced visual intelligibility. Theencrypted video data may produce a default image in the affected imagearea, which may be for example, random pixel colors.

FIG. 6F illustrates a reduced mode, in which the color depth levels ofthe image area can be set such that on playback the image area isvisually unintelligible or of reduced visual intelligibility.

In some embodiments, the obscured recording modes may include one ormore blurred recording modes, in which video data is blurred such thaton playback the image area is visually unintelligible or of reducedvisual intelligibility. As used herein, blurring refers to the reductionof high-frequency content in the image area so as to visually obscurethe image area. Such blurring may be achieved by controlling thecontrast/resolution of the image area, applying a low-pass filter,adjusting the image data compression, subsampling, remapping pixel colorlevels into a compressed range, or by any other method known toselectively blur image data. In at least one embodiment, blurring isaccomplished via replacing a pixel value with the average of itsoriginal value and the values of adjacent pixels, e.g., the nineadjacent pixels in a 3×3 matrix centered on the averaged pixel. Thisaveraging reduces the local contrast and high-frequency content of theimage, blurring it.

As discussed above, normal and obscured recording modes may berespectively enabled for different image areas or areas-of-interest.Thus, the recording modes may also include such arrangements of thenormal and obscured recording modes within the image. For example, FIG.6B illustrates a bunk-obscured mode in which the bunk area of the cabinis obscured while the driver and passenger seats are unobscured. Assuch, the privacy of bunk area users is maintained. Alternatively, FIG.6G illustrates a peripheral vision mode, in which the regions of thedoors are unobscured while everything between those regions is obscured.In the peripheral vision mode, the camera may also function as a motionsensor, as discussed herein. As such, the camera may monitor for thedoor(s) opening or remaining open, the windows being lowered orremaining lowered, etc.

Moreover, as shown for example in FIGS. 6C and 6F, the areas-of-interestcan also be defined with respect to the driver and/or passenger, orportions thereof, including but not limited to faces and other bodyparts. In other words, obscuration may be locally tied to a vehicleoccupant rather than cabin region. It will be understood, however, thatsome obscuration may be tied to the occupant, whereas other obscurationmay be tied to cabin region.

In a split stream recording mode (not shown), the video data isseparated by one or more defined characteristics and/or features. Forexample, high-frequency image content (i.e., fine detail image data) maybe stored separately from low-frequency image content (i.e., coarsedetail image data). For example, the recorder can average the values inn×n blocks of pixels, producing the low-frequency image for storage. Thepixel-wise deviations from the n×n block averages may be separatelystored as the high-frequency image content. An obscured video (or imagearea thereof) can therefore be replayed via the low-frequency imagecontent, whereas an un-obscured video (or image area thereof) can beachieved by combining the high-frequency image content with thelow-frequency image content. Alternatively, the all-frequency imagecontent may be stored separately from low-frequency image content. Anobscured video (or image area thereof) can therefore be replayed via thelow-frequency image content, whereas an un-obscured video (or image areathereof) can be replayed via the all-frequency image content. Thehigh-frequency/all-frequency image content may also be encrypted orotherwise access-restricted, so as to further facilitate driver privacywhile permitting legitimate administrator access to the videorecordings.

The split stream recording mode may separate the video data according toother characteristics and/or features, including but not limited to:face versus non-face areas, moving versus non-moving areas, etc. In thismanner, for example, the spit stream recording mode may produce aplayback image of a moving outline of a person with obscured internalcontent (e.g., darkened out or blurred) in an otherwise unobscuredenvironment.

Obscuration may also vary with time and/or space. For example, thedegree of blurring may be governed by the mathematical filtering that isapplied to the image area. There is more blurring if a larger block ofpixels is averaged together to replace the central value. In someembodiments, this block size can be varied over time or space. Near adriver's face or other location of interest, for example, a spatiallyextensive averaging can be applied, whereas far from such privacysensitive areas, a less spatially extensive blurring can be applied. Thespatially varying blur produces a ‘things are sharp away from the driverbut blurred when near to him’ effect. Moreover, when a triggering eventoccurs, less blurring can be applied close in time to the event, whereasmore blurring can be applied further in time from the event. Thistime-varying blur produces a ‘things come into focus when it happens’effect.

It will be understood that other cameras may be operated according tothe aforementioned recording modes, and that additional recording modesmay be available for both driver facing cameras and other interiorand/or exterior cameras. Other available modes may include: a full cabinmode, a short recording mode, a film mode, and a snapshot series mode, along exposure mode, a defocus mode, a random time replacement mode, atime-varying random displacement of pixels mode, a color-pane-swap mode,etc.

The processor may switch the system between different recording modesbased on the detection of one or more mode-change events, which may bedetermined based on, for example, detected driver activity, vehiclestate, telematics information, and/or safety events. Exemplarymode-change events may include, for example and without limitation,ignition on/off, engine on/off, parking brake engaged/disengaged, gearin driver/park, driver seat occupied/unoccupied, bunk areaoccupied/unoccupied, occupant face recognized/unrecognized, telematicslogged-in/logged-out, etc. For example, switching to an unobscured modemay be done in response to one or more of the following: a safety eventoccurs, the driver is detected in the driver's seat and the bunk isempty, the bunk is vacated, a passenger is detected entering thepassenger seat, and the vehicle telematics system is logged into.

It will also be understood that the aforementioned description considerssituations where the recorders are in an “on” state. However, it isfurther contemplated that the recorders may, individually orcollectively, be in an “off” or “idle” state, in which no recordingtakes place. Transitioning between the “off” or “idle” states and the“on” state can be triggered by detected events, such as safety events,non-safety events and theft events.

In some embodiments, the recording mode is adjusted to a low-power,situation driven recording mode in response to the detection of theengine as being “off.” In the low-power, situation driven recordingmode, the recording may occur in response to a triggering event, e.g. atheft or safety event, while the engine is “off,” and the recorded videodata may be stored locally. The recorded video data may be transmittedto the server when the engine is restarted.

Referring now to FIGS. 2-3, an operation of the obscuring/blurring ofvideo data in accordance with the principles of the invention will nowbe discussed.

The operation begins at block 310, with the camera 145 generating videodata of the captured image area. The video data may be captured on acontinuous basis, or in response to a detected event. In certainembodiments, such data may comprise a sequence of video frames withseparate but associated sensor data that has been collected from one ormore on-vehicle sensors or devices, as detailed herein.

At block 320, the video data may be stored in a delay buffer 242. Thedelay buffer 242 provides a time delay from the capturing of the videodata before it is recorded to the DVR memory 140-b via one of data paths216 and 218, at block 340.

The state of switch 236 determines which of the data paths the videodata takes to DVR memory 140-b, where it is stored. Where the switch isset to an un-obscured mode, the video data takes data path 216, whichstores the video data in the DVR memory 140-b without being obscured.Where the switch is set to an obscuring mode, the video data takes datapath 218, which sends the video data to an obscuring module 238, whereit is partially or wholly obscured in accordance with the principlesdiscussed herein, prior to storage in the DVR memory.

The control of the switching between obscuring modes will now bediscussed. At block 310, the sensor data is generated by the one or moreon-vehicle sensors or devices 114. At block 330, the sensor data isevaluated 232 via the processor 130 for the occurrence of an obscuringevent, e.g., the detection of an occupant in the bunk area.

If the obscuring event is detected, a obscuring-mode control signal 210is sent to the switch, at block 340. The control signal switches thedata path of the video data exiting the delay buffer 242 from data path216, which may be the default, to data path 218 and the obscuring module238. The obscuring-mode control signal 210 is also sent to a timer 234to reset the timer to a max count.

If no obscuring event is detected, either because the event has ended orhas not started, a countdown control signal 212 is sent to the timer 234to start a countdown from the max count. When the timer 234 counts downto zero, a reset signal 214 is sent to the switch 236, which resets theswitch 236 so that video data from the delay buffer 242 travels alongdata path 216.

The durations of the delay buffer 238 and the max count determine theunobscured prequel and sequel lengths to the recording of the obscuredevent. As such, the video data can be obscured in close to real time,with appropriate prequel and sequel periods.

The obscuring module 238 can obscure the video data, wholly or in part,by any method of obscuring video data, including those methods discussedherein. The video data can be obscured directly, in which the video datais recorded to the memory 140 as obscured video data. The video data canalso be obscured indirectly, in which case the obscuring module 238 addsor otherwise embeds one or more flags or other indications to the videodata instructing a playback module (not shown) to appropriately obscurethe flagged video data during playback. Such flags may identify to theplayback module which parts of the video image to obscure, as well aswhen to start/stop the obscuring. In one or more embodiments, theobscuring module 238 may be configured to add a value set to the videodata. The value set may indicate a degree, type, and local image areacoverage for an obscuration region to be applied to the video data andused to obscure the video data at the off-vehicle server.

It will be understood that while the illustrative altering of the videodata is described in terms of obscuring the video data, the video datacan be blurred or altered in other ways for driver privacy, as discussedherein.

The playback of the recorded video may also be affected by access levelsassociated with the video data and/or the obscuration. The obscuredvideo data may be played back with image areas obscured by defaultoperation. However, one or more of the obscured image areas may beassociated with a corresponding access level or encryption key. As such,the one or more of the obscured image areas may be “revealed” inaccordance with provided access rights of the viewing party or theproviding of the encryption key via a video player. For example, lawenforcement, insurance companies, and/or driver may have access to viewobscured image areas. The various obscured images may be associated withcorresponding access levels based on the detected circumstances thattrigger the obscuration. Moreover, the access levels may be stored inthe metadata of the video data.

In some embodiments, a degree of obscuration, including non-obscuration,may be stored in the metadata of the video data and may be adjustableduring playback. Accordingly, the timestamped detected circumstances maybe correlated with the degree of obscuration such that the degree ofobscuration for an image frame or series of image frames can be adjustedbased on the type of detected circumstances associated with the imageframe(s). For example, facial obscuration may be eliminated on playback,whereas bunk area obscuration may not—or vice versa. In this manner,just as the degree, type, appropriate circumstances and conditions ofthe obscuration applied to the recording may be set, so too can thedegree, type, relevant circumstances and person-specificities of therevealed recording be set.

It will further be appreciated that the video and non-video data may betimestamped based on a central or common clock such that the video andnon-video data can be cross-referenced using the timestamps. Similarly,embedded tags within the video and/or non-video data may be used tocorrelate the data sets. Additionally, non-video data may be correlatedto video data by storing the non-video data in the frame headers. One ormore techniques for storing video and non-video data in an associatedfashion, whether in a single memory or in separate memories, aredescribed in detail in U.S. application Ser. No. 16/208,375, entitled“System and Method for Providing Complete Event Data fromCross-Referenced Data Memories,” which is assigned to the assigneehereof, and the entire disclosure of which is hereby incorporated byreference.

An audio and/or visual indicator, such as for example, a small light onthe dashboard or in the cabin, may also be provided to indicate thatrecording is in progress. Accordingly, the indicator may be operativelycoupled to the processor and responsive to processor commands to turnon/off in connection with the recording.

The principles of the invention may also be applied to audio recordingscaptured by one or more microphones, independently or in connection withthe cameras.

Turning now to FIGS. 3-4, the principles of the invention areillustrated as applied to the video-clip recordings. The operation issubstantially similar to the operation discussed with reference to FIGS.2-3, and for the sake of brevity will only be discussed briefly with theunderstanding that one of ordinary skill in the art will readilyappreciate the similarities without undue and duplicative description.

At block 310, the sensor data is evaluated 432, via the processor, forthe occurrence of a recording triggering event, e.g., a safety event. Ifthe recording triggering event is detected, a video-clip control signal410 is sent to switch 436 at block 340. The video-clip control signal410 allows the video data to be sent to the data paths 216 and 218 to berecorded, whereas the default state of switch 436 prevents recording ofthe video data. A video-clip timer 434 may also be utilized with thedelay buffer 242, in similar manner to timer 234, to determine theunobscured prequel and sequel lengths to the recording of the recordingtriggering event.

The principles of the invention may also be applied to both thevideo-clip recorder and the DVR recorder in tandem, as shown for examplein FIG. 5. Here, the sensor data is provided to a combined evaluationunit 520 that controls a combined switching unit 540, which in turndirects the video data from the delay buffer 242 to a state of not beingrecorded 140-c, or to the appropriate memory 140-a, 140-b via theobscuring module 238 or not, as appropriate in accordance with theprinciples discussed herein.

It should further be appreciated that correlated obscured and unobscureddata may be transmitted by the on-vehicle event detection and reportingsystem to a remote server using, for example, wireless transceiver 150of FIG. 1. This transmission of event data may be done automatically inresponse to the occurrence of a detected event, or on request.

Once the correlated event data has been transmitted from the on-vehicleevent detection and reporting system to the remote server, whetherautomatically or upon request, the data may be provided to a user-sideviewing portal or application. In certain embodiments, the viewingportal may comprise a web application provided on a user computer, suchas the web application. It should further be appreciated that theviewing portal or application may be a dedicated software programexecuting on a computer or mobile device that is accessible to the user.Regardless of how the data is made available to a user-side device, theuser will be able to view or otherwise discern only the unobstructedvideo data, while the details of the obscured video data remain visuallyobscured. And, since there are no data gaps despite at least some of thevideo data being obscured, the user will be able to engage in all of thesame post-processing and data analysis, which is not otherwise dependenton the specific details of the obscured data.

It should further be appreciated that the system may further beconfigured to detect when obscuring image areas is no longer necessaryand, in response thereto, automatically exit the obscuring mode(s) sothat regular video data capturing and recording is resumed.

As used herein, the terms “camera” or “cameras” are intended to refer toany and all digital imaging devices, including but not limited tocameras. Moreover, references to “driver,” “passenger,” and “occupant,”should be understood to be interchangeable, and the principles of theinvention understood to apply as appropriate to each.

As used herein, the terms “a” or “an” shall mean one or more than one.The term “plurality” shall mean two or more than two. The term “another”is defined as a second or more. The terms “including” and/or “having”are open ended (e.g., comprising). The term “or” as used herein is to beinterpreted as inclusive or meaning any one or any combination.Therefore, “A, B or C” means “any of the following: A; B; C; A and B; Aand C; B and C; A, B and C”. An exception to this definition will occuronly when a combination of elements, functions, steps or acts are insome way inherently mutually exclusive.

Reference throughout this document to “one embodiment”, “certainembodiments”, “an embodiment” or similar term means that a particularfeature, structure, or characteristic described in connection with theembodiment is included in at least one embodiment of the presentinvention. Thus, the appearances of such phrases or in various placesthroughout this specification are not necessarily all referring to thesame embodiment. Furthermore, the particular features, structures, orcharacteristics may be combined in any suitable manner on one or moreembodiments without limitation.

In accordance with the practices of persons skilled in the art ofcomputer programming, the invention is described herein with referenceto operations that are performed by a computer system or a likeelectronic system. Such operations are sometimes referred to as beingcomputer-executed. It will be appreciated that operations that aresymbolically represented include the manipulation by a processor, suchas a central processing unit, of electrical signals representing databits and the maintenance of data bits at memory locations, such as insystem memory, as well as other processing of signals. The memorylocations where data bits are maintained are physical locations thathave particular electrical, magnetic, optical, or organic propertiescorresponding to the data bits.

The term “server” means a functionally-related group of electricalcomponents, such as a computer system that may or may not be connectedto a network and which may include both hardware and softwarecomponents, or alternatively only the software components that, whenexecuted, carry out certain functions. The “server” may be furtherintegrated with a database management system and one or more associateddatabases.

In accordance with the descriptions herein, the term “computer readablemedium,” as used herein, refers to any non-transitory media thatparticipates in providing instructions to the processor 130 forexecution. Such a non-transitory medium may take many forms, includingbut not limited to volatile and non-volatile media. Non-volatile mediaincludes, for example, optical or magnetic disks. Volatile mediaincludes dynamic memory for example and does not include transitorysignals, carrier waves, or the like.

In addition and further in accordance with the descriptions herein, theterm “logic,” as used herein, with respect to FIG. 1, includes hardware,firmware, software in execution on a machine, and/or combinations ofeach to perform a function(s) or an action(s), and/or to cause afunction or action from another logic, method, and/or system. Logic mayinclude a software controlled microprocessor, a discrete logic (e.g.,ASIC), an analog circuit, a digital circuit, a programmed logic device,a memory device containing instructions, and so on. Logic may includeone or more gates, combinations of gates, or other circuit components.

The foregoing disclosure has been set forth merely to illustrate theinvention and is not intended to be limiting. Since modifications of thedisclosed embodiments incorporating the spirit and substance of theinvention may occur to persons skilled in the art, the invention shouldbe construed to include everything within the scope of the appendedclaims and equivalents thereof.

What is claimed is:
 1. A system for recording vehicle occupants andtheir immediate environment, comprising: a driver-facing camera forcapturing video data; an input bus that receives data from one or moresensors configured to detect vehicle related data; and a processorconfigured to embed, based on the sensor data, at least one indicator ina portion of the video data, wherein the indicator designates that theportion of the video data is to be obscured during playback of the videodata, wherein a first indicator identifies a start time for a playbackmodule to begin obscuring the video data and a second indicatoridentifies a stop time for the playback module to end obscuring thevideo data, wherein setting the parking brake identifies a start timeand detection of movement in the vicinity of the exterior of the vehicleidentifies a stop time.
 2. The system of claim 1, wherein setting theparking brake of the vehicle embeds the at least one indicator in theportion of the video data.
 3. The system of claim 1, wherein theplayback module includes a gradual transition to unobscured video dataafter the stop time.
 4. The system of claim 1, wherein the at least oneindicator includes a value set to indicate at least one of a degree, atype, and a local image area for a region to be obscured in the videodata by a playback module.
 5. The system of claim 1, wherein obscuring aportion of the video data during playback includes changing at least oneof: frame rate, frame resolution, pixel colors, local defocus degree,contrast, detail level, compression ratio, pixel order, icon or fixedsnapshot substitution, recording time, pixel content, exposure,brightness, obscuration boundaries, and recording activity.
 6. Thesystem of claim 1, wherein the vehicle related data includes one or moreof: time, driver or passenger location in the vehicle, vehicle behavior,movement detection, parking brake status and ignition status.
 7. Thesystem of claim 1, further comprising: a transceiver configured totransmit the video data having the at least one indicator to anoff-vehicle server, the off-vehicle server associated with a playbackmodule.
 8. The system of claim 1, further comprising a playback modulefor playing back the video data, wherein the playback module includesthe functionality to obscure the video data in response to the at leastone indicator.
 9. The system of claim 1, wherein the at least oneindicator includes an access rights set, such that a user having amatching access rights set can see the video data without obscuration.10. A method for recording vehicle occupants and their immediateenvironment, comprising: receiving vehicle-related data from one or moresensors; receiving video data from a driver-facing camera; embedding atleast one indicator in a portion of the video data based on the sensordata, wherein the indicator designates that the portion of the videodata is to be obscured during playback of the video data; embedding afirst indicator identifying a start time for a playback module to beginobscuring of the video data; and embedding a second indicatoridentifying a stop time for the playback module to end obscuring of thevideo data, wherein setting the parking brake identifies a start timeand detection of movement in the vicinity of the exterior of the vehicleidentifies a stop time.
 11. The method of claim 10, further comprising:transmitting the video data having the at least one indicator to anoff-vehicle server, the off vehicle server having an associated playbackmodule.
 12. The method of claim 10, further comprising: playing back thevideo data on a playback module, wherein the playback module obscuresthe video data portion associated with the at least one indicator. 13.The method of claim 10, further comprising embedding an access rightsset, such that a user having a matching access rights set can see thevideo data without obscuration.